<!DOCTYPE html>
<html lang="zh">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
        <title>cs229 第六节 | Dirac Lee</title><meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="robots" content="noodp" />
<meta name="Description" content="cs229 第六节"><link rel="prev" href="https://diraclee.gitee.io/2020/07/install_manjaro_kde/" /><link rel="next" href="https://diraclee.gitee.io/2020/10/cs229_lecture7/" /><link rel="canonical" href="https://diraclee.gitee.io/2020/10/cs229_lecture6/" />
<link rel="shortcut icon" type="image/x-icon" href="/favicon.ico" />
<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png">
<link rel="manifest" href="/site.webmanifest">
<link rel="mask-icon" href="/safari-pinned-tab.svg" color="#5bbad5">
<meta name="msapplication-TileColor" content="#da532c">
<meta name="theme-color" content="#ffffff"><meta property="og:title" content="cs229 第六节" />
<meta property="og:description" content="cs229 第六节" />
<meta property="og:type" content="article" />
<meta property="og:url" content="https://diraclee.gitee.io/2020/10/cs229_lecture6/" />
<meta property="article:published_time" content="2020-10-14T00:00:00+00:00" />
<meta property="article:modified_time" content="2020-10-14T00:00:00+00:00" />
<script type="application/ld+json">
    {
        "@context": "http://schema.org",
        "@type": "BlogPosting",
        "headline": "cs229 第六节",
        "mainEntityOfPage": {
            "@type": "WebPage",
            "@id": "https:\/\/diraclee.gitee.io\/2020\/10\/cs229_lecture6\/"
        },"image": {
                "@type": "ImageObject",
                "url": "https:\/\/diraclee.gitee.io\/logo.png",
                "width":  800 ,
                "height":  600 
            },"genre": "posts","keywords": "机器学习, cs229","wordcount":  1276 ,
        "url": "https:\/\/diraclee.gitee.io\/2020\/10\/cs229_lecture6\/","datePublished": "2020-10-14T00:00:00\x2b00:00","dateModified": "2020-10-14T00:00:00\x2b00:00","license": "This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.","publisher": {
                "@type": "Organization",
                "name": "xxxx",
                "logo": {
                "@type": "ImageObject",
                "url": "https:\/\/diraclee.gitee.io\/logo.png",
                "width":  127 ,
                "height":  40 
                }
            },"description": "cs229 第六节"
    }
    </script><link rel="stylesheet" href="/css/style.min.css"><link rel="stylesheet" href="/css/lib/fontawesome-free/all.min.css"><link rel="stylesheet" href="/css/lib/animate/animate.min.css"></head>
    <body><script>
            window.isDark = (window.localStorage && window.localStorage.getItem('theme')) === 'dark';
            window.isDark && document.body.classList.add('dark-theme');
        </script><div class="wrapper"><nav class="navbar">
    <div class="navbar-container">
        <div class="navbar-header animated bounceIn">
            <a href="https://diraclee.gitee.io">Dirac Lee</a>
        </div>
        <div class="navbar-menu"><a class="menu-item" href="https://diraclee.gitee.io/posts" title="">文章</a><a class="menu-item" href="https://diraclee.gitee.io/tags" title="">标签</a><a class="menu-item" href="https://diraclee.gitee.io/categories" title="">分类</a><a class="menu-item" href="https://diraclee.gitee.io/about" title="">关于</a><a class="menu-item" href="https://diraclee.gitee.io" title="English"><i class="fas fa-language fa-fw"></i></a><a href="javascript:void(0);" class="theme-switch"><i class="fas fa-adjust fa-rotate-180 fa-fw" title="切换主题"></i></a>
        </div>
    </div>
</nav><nav class="navbar-mobile">
    <div class="navbar-container">
        <div class="navbar-header">
            <div class="navbar-header-title animated bounceIn">
                <a href="https://diraclee.gitee.io">Dirac Lee</a>
            </div>
            <div class="menu-toggle" id="menu-toggle">
                <span></span><span></span><span></span>
            </div>
        </div>
        <div class="navbar-menu" id="mobile-menu"><a class="menu-item" href="https://diraclee.gitee.io/posts" title="">文章</a><a class="menu-item" href="https://diraclee.gitee.io/tags" title="">标签</a><a class="menu-item" href="https://diraclee.gitee.io/categories" title="">分类</a><a class="menu-item" href="https://diraclee.gitee.io/about" title="">关于</a><a class="menu-item" href="https://diraclee.gitee.io" title="English"></a><a href="javascript:void(0);" class="theme-switch"><i class="fas fa-adjust fa-rotate-180 fa-fw" title="切换主题"></i></a>
        </div>
    </div>
</nav><main class="main">
                <div class="container"><article class="page"><h1 class="post-title animated flipInX">cs229 第六节</h1><div class="post-meta">
            <div class="post-meta-main"><a class="author" href="https://diraclee.gitee.io" rel="author" target="_blank">
                    <i class="fas fa-user-circle fa-fw"></i>Dirac Lee
                </a>&nbsp;<span class="post-category">收录于&nbsp;<i class="far fa-folder fa-fw"></i><a href="https://diraclee.gitee.io/categories/%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0/">学习笔记</a>&nbsp;</span></div>
            <div class="post-meta-other"><i class="far fa-calendar-alt fa-fw"></i><time datetime=2020-10-14>2020-10-14</time>&nbsp;
                <i class="fas fa-pencil-alt fa-fw"></i>约 1276 字&nbsp;
                <i class="far fa-clock fa-fw"></i>预计阅读 3 分钟&nbsp;</div>
        </div><div class="post-content"><p>生成学习算法 (Generative Learning Algorithm)</p>
<ul>
<li>高斯判别分析 (Gaussian Discriminant Analysis)</li>
<li>生成 &amp; 判别算法比较</li>
<li>朴素贝叶斯</li>
</ul>
<a class="post-dummy-target" id="引论"></a><h2>引论</h2>
<p><figure><img src="/svg/loading.min.svg" data-sizes="auto" data-src="https://gitee.com/DiracLee/picbed/raw/master/img/20201014202629.png" alt="" class="lazyload"></figure></p>
<a class="post-dummy-target" id="判别算法"></a><h3>判别算法</h3>
<p>如：逻辑回归</p>
<p>x -&gt; y</p>
<p>学习 P(y | x) 或 直接学习 $h_\theta (x) = \begin{cases} 0 \\ 1 \end{cases}$</p>
<a class="post-dummy-target" id="生成算法"></a><h3>生成算法</h3>
<p>学习 P(x | y)  和 P(y)</p>
<p>x 是特征， y 是类别</p>
<p>P(y) 是先验类别，即在我们看到特征之前预估的类别</p>
<a class="post-dummy-target" id="贝叶斯公式"></a><h3>贝叶斯公式</h3>
<p>$$
P(y = 1 | x) = \frac {P(x | y = 1) P(y = 1)} {P(x)}
$$</p>
<p>其中</p>
<p>$$
P(x) = P(x | y = 1) P(y = 1) + P(x | y = 0) P(y = 0)
$$</p>
<a class="post-dummy-target" id="高斯判别分析-gda"></a><h2>高斯判别分析 (GDA)</h2>
<p>假设 $x \in \R^n$ (不再添加额外特征 $x_0 = 1$)</p>
<p>假设 P(x | y) 是高斯分布，记 $z = x | y$ 则  $z \sim N(\vec \mu, \Sigma)$</p>
<p>其中 $\vec \mu \in \R^n$ 是均值向量，$\Sigma \in \R^{n \times n}$ 是协方差矩阵</p>
<p>即 $E[z] = \vec \mu$，$Cov[z] = E[(z - \vec \mu) (z - \vec \mu)^T] = E[z z^T] - E_z E_z^T$</p>
<p>$$
P(z) = \frac 1 {(2 \pi)^{\frac n 2} |\Sigma|^{\frac 1 2}} \exp [ - \frac 1 2 (z - \vec \mu)^T \Sigma^{-1} (z - \vec \mu) ]
$$</p>
<p>![二维高斯图]</p>
<p>![二维高斯俯视图]</p>
<a class="post-dummy-target" id="gda-模型"></a><h3>GDA 模型</h3>
<p>$$
P(x | y = 0) = \frac 1 {(2 \pi)^{\frac n 2} |\Sigma|^{\frac 1 2}} \exp [ - \frac 1 2 (z - \vec \mu_0)^T \Sigma^{-1} (z - \vec \mu_0) ]
$$</p>
<p>$$
P(x | y = 1) = \frac 1 {(2 \pi)^{\frac n 2} |\Sigma|^{\frac 1 2}} \exp [ - \frac 1 2 (z - \vec \mu_1)^T \Sigma^{-1} (z - \vec \mu_1) ]
$$</p>
<p>由于以上两个类别的协方差矩阵 $\Sigma$ 相同，因此我们会得到线性判定边界。</p>
<p>$$
\begin{aligned}
P(y) = \phi^y (1 - \phi)^{1 - y} = 
\begin{cases}
\phi      &amp;&amp;  if &amp;&amp; y = 1 \\ 
1 - \phi  &amp;&amp;  if &amp;&amp; y = 0
\end{cases}
\end{aligned}
$$</p>
<a class="post-dummy-target" id="训练集"></a><h3>训练集</h3>
<p>$$
\lbrace (x^{(i)}, y^{(i)})  \rbrace_{i=1}^{m}
$$</p>
<p>联合可能性 (Joint Likelihood)</p>
<p>$$
\begin{aligned}
\mathscr L(\phi, \mu_1, \mu_2, \Sigma) 
&amp;= \prod_{i=1}^m P(x^{(i)}, y^{(i)}; \phi, \mu_0, \mu_1, \Sigma) \\ 
&amp;= \prod_{i=1}^m P(x^{(i)}| y^{(i)}) P(y^{(i)})
\end{aligned}
$$</p>
<blockquote>
<p>注：判别算法条件可能性 $\mathscr L(\theta) = \prod_{i=1}^m P(y^{(i)} | x^{(i)}; \theta)  $</p>
</blockquote>
<p>对数联合可能性</p>
<p>$$
l(\phi, \mu_1, \mu_2, \Sigma) = \log \mathscr L(\phi, \mu_1, \mu_2, \Sigma) 
$$</p>
<a class="post-dummy-target" id="最大可能性评估"></a><h3>最大可能性评估</h3>
<p>$$
\begin{aligned}
\arg \max_{\phi, \mu_0, \mu_1, \Sigma} l(\phi, \mu_1, \mu_2, \Sigma)
\end{aligned}
$$</p>
<p>解得</p>
<p>$$
\phi = \frac { \sum_{i=1}^m y^{(i)} } m = \frac { \sum_{i=1}^m I(y^{(i)} = 1) } m
$$</p>
<p>$$
\mu_0 = \frac {\sum_{i=1}^m I(y^{(i)} = 0) x^{(i)}} {\sum_{i=1}^m I(y^{(i)} = 0) }
$$</p>
<p>$$
\mu_1 = \frac {\sum_{i=1}^m I(y^{(i)} = 1) x^{(i)}} {\sum_{i=1}^m I(y^{(i)} = 1) }
$$</p>
<p>$$
\Sigma = \frac 1 m \sum_{i=1}^m (x^{(i)} - \mu_{y^{(i)}}) (x^{(i)} - \mu_{y^{(i)}})^T
$$</p>
<a class="post-dummy-target" id="预测"></a><h3>预测</h3>
<p>$$
\arg \max_y P(y | x) = \arg \max_y \frac {P(x | y) P(y)} {P(x)} 
$$</p>
<p>由于  $P(x)$ 只是一个与 y 无关的正数，因此求 $\arg \max$ 时无需考虑，得</p>
<p>$$
\arg \max_y P(y | x) = \arg \max_y P(x | y) P(y)
$$</p>
<a class="post-dummy-target" id="生成--判别算法比较"></a><h2>生成 &amp; 判别算法比较</h2>
<p><figure><img src="/svg/loading.min.svg" data-sizes="auto" data-src="https://gitee.com/DiracLee/picbed/raw/master/img/gaussian.png" alt="" class="lazyload"></figure></p>
<p>对于确定的 $\phi$、$\mu_0$、$\mu_1$、$\Sigma$，画 $P(y = 1| x; \phi, \mu_0, \mu_1, \Sigma) $ 随 x 变化的曲线。</p>
<p><figure><img src="/svg/loading.min.svg" data-sizes="auto" data-src="https://gitee.com/DiracLee/picbed/raw/master/img/logistic.png" alt="" class="lazyload"></figure></p>
<p>刚好就是 sigmoid 函数。</p>
<a class="post-dummy-target" id="生成算法-1"></a><h3>生成算法</h3>
<p>GDA 假设（强条件假设）</p>
<p>$$
x | y = 0 \sim N(mu_0, \Sigma)
$$</p>
<p>$$
x | y = 1 \sim N(mu_1, \Sigma)
$$</p>
<p>$$
y \sim Bern(\phi)
$$</p>
<a class="post-dummy-target" id="判别算法-1"></a><h3>判别算法</h3>
<p>逻辑回归假设（弱条件假设）</p>
<p>$$
P(y = 1 | x) = \frac 1 { 1 + e^{-\theta^T x} }
$$</p>
<blockquote>
<p>前者可以推出后者，但后者无法推出前者</p>
</blockquote>
<p>若</p>
<p>$$
x | y = 0 \sim Posson(\lambda_0)
$$</p>
<p>$$
x | y = 1 \sim Posson(\lambda_1)
$$</p>
<p>$$
y \sim Bern(\phi)
$$</p>
<p>则 $ P(y = 1 | x)$  也符合逻辑回归。</p>
<p>实际上，只要 $P(x | y)$ 服从的分布是指数家族，$ P(y | x)$ 都符合逻辑回归。</p>
<blockquote>
<p>如果我们有很多数据，那就用逻辑回归，因为假设条件更弱，适用性更广。如果可以确定数据符合高斯分布，那么可以选用 GDA。模型从两个渠道进行学习：一是提供的数据，而是预制的假设。</p>
</blockquote>
<a class="post-dummy-target" id="朴素贝叶斯"></a><h2>朴素贝叶斯</h2>
<p>场景：邮件过滤</p>
<p>问题：特征 x 如何设计？</p>
<p>on-hot 编码：特征的每个维度代表一个单词，在邮件中出现了就取 0，否则取 1.</p>
<p>$$
x \in \lbrace 0, 1 \rbrace^n 
$$</p>
<p>$$
x_j = I(第 i 个单词出现在邮件中)
$$</p>
<p>目标是对 $P(x | y)$ 和 $ P(y) $ 进行建模，x 共有 $2^n$ 种可能取值。</p>
<p>$$
P(x_1, x_2, &hellip; x_n | y) = P(x_1 | y) P(x_2 | x_1, y) P(x_3 | x_2, x_1, y)&hellip; P(x_n | x_{n-1}, x_{n-2}, &hellip;, x_1, y)
$$</p>
<p>假设对给定的 y， $x_j$ 之间是条件独立的，则</p>
<p>$$
P(x_1, x_2, &hellip; x_n | y) = P(x_1 | y) P(x_2 | y) P(x_3 | y)&hellip; P(x_n | y) = \prod_{i = 1}^n P(x_i | y)
$$</p>
<p>记号</p>
<p>$$
\phi_j | y = 1 = P(x_j = 1 | y = 1)
$$</p>
<p>$$
\phi_j | y = 0 = P(x_j = 1 | y = 0)
$$</p>
<p>$$
\phi_y = P(y = 1)
$$</p>
<p>联合可能性</p>
<p>$$
\mathscr L(\phi_y, \phi_{i | y}) = \prod_{i = 1}^m P(x^{(i)}, y^{(i)}; \phi_y, \phi_{j | y})
$$</p>
<p>MLE</p>
<p>$$
\phi_y = \frac { \sum_{i = 1}^m I(y^{(i)} = 1) } m
$$</p>
<p>$$
\phi_{j | y = 1} = \frac { \sum_{i = 1}^m I( x^{(i)}_j = 1, y^{(i)} = 1) } { \sum_{i = 1}^m I( y^{(i)}) = 1 }
$$</p>
<a class="post-dummy-target" id="参考"></a><h2>参考</h2>
<p><a href="https://katex.org/docs/supported.html" target="_blank">$\KaTeX$ 文档</a></p></div><div class="post-footer" id="post-footer">
    <div class="post-info">
        <div class="post-info-line">
            <div class="post-info-mod">
                <span>本文于 2020-10-14 更新</span>
            </div>
            <div class="post-info-license"></div>
        </div>
        <div class="post-info-line">
            <div class="post-info-md"></div>
            <div class="post-info-share"><span><a href="//www.linkedin.com/shareArticle?url=https%3a%2f%2fdiraclee.gitee.io%2f2020%2f10%2fcs229_lecture6%2f&amp;title=cs229%20%e7%ac%ac%e5%85%ad%e8%8a%82" target="_blank" title="分享到 LinkedIn">
            <i class="fab fa-linkedin fa-fw"></i>
        </a><a href="//service.weibo.com/share/share.php?url=https%3a%2f%2fdiraclee.gitee.io%2f2020%2f10%2fcs229_lecture6%2f&amp;appkey=&amp;title=cs229%20%e7%ac%ac%e5%85%ad%e8%8a%82" target="_blank" title="分享到 Weibo">
            <i class="fab fa-weibo fa-fw"></i>
        </a></span></div>
        </div>
    </div>

    <div class="post-info-more">
        <section><span class="tag">
                        <a href="https://diraclee.gitee.io/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"><i class="fas fa-tag fa-fw"></i>&nbsp;机器学习</a>&nbsp;
                    </span><span class="tag">
                        <a href="https://diraclee.gitee.io/tags/cs229/"><i class="fas fa-tag fa-fw"></i>&nbsp;cs229</a>&nbsp;
                    </span></section>
        <section>
            <span><a href="javascript:window.history.back();">返回</a></span>&nbsp;|&nbsp;<span><a href="https://diraclee.gitee.io">主页</a></span>
        </section>
    </div>

    <div class="post-nav"><a href="https://diraclee.gitee.io/2020/07/install_manjaro_kde/" class="prev" rel="prev" title="Manjaro KED 配置全攻略(转)"><i class="fas fa-angle-left fa-fw"></i>Manjaro KED 配置全攻略(转)</a>
            <a href="https://diraclee.gitee.io/2020/10/cs229_lecture7/" class="next" rel="next" title="cs229 第七节">cs229 第七节<i class="fas fa-angle-right fa-fw"></i></a></div>
</div><div class="post-comment"></div>
    </article></div>
            </main><footer class="footer">
    <div class="copyright"><div class="copyright-line">由 <a href="https://gohugo.io/" target="_blank" rel="external nofollow noopener noreffer">Hugo</a> 强力驱动 | 主题 - <a href="https://github.com/dillonzq/LoveIt" target="_blank" rel="external nofollow noopener noreffer">LoveIt<i class="far fa-heart fa-fw"></i></a>
        </div>

        <div class="copyright-line"><i class="far fa-copyright fa-fw"></i><span itemprop="copyrightYear">2020</span><span class="author" itemprop="copyrightHolder">&nbsp;<a href="https://diraclee.gitee.io" target="_blank">Dirac Lee</a></span>&nbsp;|&nbsp;<span class="license"><a rel="license external nofollow noopener noreffer" href="https://creativecommons.org/licenses/by-nc/4.0/" target="_blank">CC BY-NC 4.0</a></span></div>
    </div>
</footer></div><a href="#" class="dynamic-to-top" id="dynamic-to-top" data-scroll>
            <span>&nbsp;</span>
        </a><script src="/js/lib/jquery/jquery.slim.min.js"></script><script src="/js/lib/lazysizes/lazysizes.min.js"></script><script src="/js/lib/smooth-scroll/smooth-scroll.polyfills.min.js"></script><script>window.scroll = new SmoothScroll('[data-scroll]', {speed: 300, speedAsDuration: true});</script><link rel="stylesheet" href="/css/lib/katex/katex.min.css"><script src="/js/lib/katex/katex.min.js"></script><script defer src="/js/lib/katex/auto-render.min.js"></script><link rel="stylesheet" href="/css/lib/katex/copy-tex.min.css"><script defer src="/js/lib/katex/copy-tex.min.js"></script><script defer src="/js/lib/katex/mhchem.min.js"></script><script>
        document.addEventListener("DOMContentLoaded", function () {
            renderMathInElement(document.body, {
                delimiters: [
                    { left: "$$", right: "$$", display: true },
                    { left: "\\(", right: "\\)", display: false },
                    { left: "\\[", right: "\\]", display: true },{ left: "$", right: "$", display: false },]
            });
        });
    </script><script src="/js/blog.min.js"></script></body>
</html>